General Disclaimer 


One or more of the Following Statements may affect this Document 


• This document has been reproduced from the best copy furnished by the 
organizational source. It is being released in the interest of making available as 
much information as possible. 


• This document may contain data, which exceeds the sheet parameters. It was 
furnished in this condition by the organizational source and is the best copy 
available. 


• This document may contain tone-on-tone or color graphs, charts and/or pictures, 
which have been reproduced in black and white. 


• This document is paginated as submitted by the original source. 


• Portions of this document are not fully legible due to the historical nature of some 
of the material. However, it is the best reproduction available from the original 
submission. 


Produced by the NASA Center for Aerospace Information (CASI) 



Image Analysis Based on Soft Computing and Applied on 
Space Shuttle Safety during the Liftoff Process. 

Jesus A. Dominguez 1 , Steve J. Klinko 2 

Imaging techniques based on Soft Computing (SC) and developed at Kennedy Space Center 
(KSC) have been implemented on a variety of prototype applications related to the safety operation 
of the Space Shuttle during the liftoff process. These SC-based prototype applications include 
detection and tracking of moving Foreign Objects Debris (FOD) during the Space Shuttle liftoff, 
visual anomaly detection on slidewires used in the emergency egress system for the Space Shuttle 
at the launch pad, and visual detection of distant birds approaching the Space Shuttle launch pad. 
This SC-based image analysis capability developed at KSC was also used to analyze images 
acquired during the accident of the Space Shuttle Columbia and estimate the trajectory and velocity 
of the foam that caused the accident. 

1. INTRODUCTION 

These imaging techniques adaptively perform image segmentation, edge extraction, and image 
enhancement using techniques developed at KSC and based on Fuzzy Reasoning (FR), one of the 
key components of SC. NASA has filed for patents on these techniques known as Fuzzy Reasoning 
Edge Detection (FRED), Fuzzy Reasoning Adaptive Threshold (FRAT), and Fuzzy Reasoning 
Image Enhancement for Binarization (FRIEB). NASA has also granted licenses of these FR-based 
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imaging techniques to companies from different fields, including surveillance, security screening, 
medical, text digitalization, and aerospace. Artificial Neural Network (ANN) and Genetic 
Algorithm (GA), two other key components of SC were also utilized to implement some of these 
prototype applications using commercially available ANN-GA software. The utilization of ANN 
and GA on image classification leads to valuable learning and optimization capabilities. 

2. SOFT COMPUTER TECHNIQUES 
2.1 Fuzzy Reasoning Edge Detection (FRED) 

Edges carry the most important information in images and accurate edge detection is vital to 
perform advanced image processing and analysis. The problem of determining what is (and what is 
not) an edge is confused by the fact that edges are very often partially hidden or distorted by 
various effects, such as inadequate lighting and image acquisition noise. Fuzzy reasoning is a 
suitable framework for expressing heuristic processes applied to incomplete and imperfect data. 
FRED uses an odd-sided window of pixels to find the edge level of the pixel in the center of the 
window via fuzzy reasoning based on a heuristic analysis of four different intensity gradients of the 
two areas formed at both sides of the four diagonals (one vertical, one horizontal, one right-inclined, 
and one left-inclined) present in the window. A crisp edge value for the pixel in the center of the 
window is finally generated after a fuzzy membership function is evaluated using the optimal 
intensity gradient. FRED autonomously determines the size of the odd-sided window used to scan 
the image based on the size of the images; the smaller window size 3x3, is set for very low 
resolution images while window sizes as high as 21x21 are set for high-resolution images. 

Figure 1 shows an image of a CD having faint scratches on the center and the edge extraction of 
this image using FRED and Sobel, a technique widely used for edge detection. 
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Figure 1. Sobel and FRED methods applied to an image containing a CD with a defect in the center. 

Figure 1 shows clearly that FRED exceeds Sobel’s performance. FRED algorithm distinctly detects 
these faint scratches while Sobel fails to detect them. Numbers and marks on the CD are much 
more evident when FRED is used. 

2.2 Fuzzy Reasoning Adaptive Thresholding (FRAT) 

Binary image processing is of special interest because an image with binary format can be 
processed with very fast logical (Boolean) operators. Since most image display systems and 
software assume images of 8 or more bits per pixel, the binarization of these images usually takes 2 
extreme gray tones, black and white, which are ordinarily represented by 0 or 255, respectively, in 
an 8-bit gray-scale display environment. 

Usually a binary image is obtained from an 8-bit gray-level image by thresholding and assigning 
the low binary (0) and high (255) values to all gray levels based on the chosen threshold. 
Obviously, the threshold that is chosen has a critical importance since it controls the binary-based 
pattern classification that is obtained from the gray-level image. The key issue is to choose an 
optimal threshold so that the number of misclassified image pixels is kept as low as possible. 



In FRAT, the image is considered as an array of singletons corresponding to image pixels, each 
having a membership value associated with a property of the pixel, the grayness level. For image 
thresholding, the membership function is defined in terms of the grade of pixel's grayness 
belonging to one of the two classes, background and foreground. The membership function is 
calculated via a triangular function at each of the two classes (background and foreground); the 
triangular function in each of the two classes is built using the average image grayness and 
histogram-based weight values at each class. In each of the two classes, the membership value 
(equal to 1) is the largest at the class average gray level and reduces its value when the difference 
between the pixel gray level and its class average level increases. This means that pixels with gray 
levels close to their corresponding class average gray levels have less fuzziness or ambiguity and 
thus can be classified with greater confidence than pixels with gray levels far from their class gray 
levels. The image entropy measure is used as a cost function to find the optimal threshold. The 
entropy factor needed to compute the entropy measure is calculated using a simple and fast 
computational linear function. 

FRAT uses a similar but more efficient and faster computational approach than the one used in 
Huang-Wang method. Huang-Wang method uses a symmetric membership function, while the 
proposed approach uses a more realistic membership function having the highest and lowest gray 
levels holding nonzero histogram values as the domain limits. Huang-Wang method also limits the 
membership values within a range of 0.5 to 1 .0 since it uses Shanon's entropy function as a cost 
function. FRAT uses a straight-line cost function that requires much less computational power than 
the Shannon function used by Huang-Wang method. Figure 2 shows the binarization of an image of 
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Figure 2. Huang-Wang and FRAT applied to an image containing a text with a corrupted background. 


a text having a corrupted and noisy background; from figure 2, it is clear that FRAT determines a 
better optimal threshold (8) than Huang-Wang (89); FRAT takes less execution time, 2.5 ms 
compared with 10.8 ms. Image thresholding is the simplest image segmentation approach. It is 
actually a pattern classification procedure in which only one input feature is involved, the pixel 
intensity value. The key issue here is to automatically choose an optimal threshold value so that the 
number of misclassified image pixels is kept as low as possible. 


2.3 Fuzzy Reasoning Image Enhancement for Binarization (FRIEB) 

One of the key goals of performing binarization of grayed-scaled images is to be able to digitize 
text and numbers contained in the grayed-scale image acquired outdoors having uncontrolled image 
acquisition conditions (lighting, camera/video stabilization, etc.). These conditions degrade the 
ending quality of the acquired gray-scaled image making more difficult to binarize the text and 
numbers contained in the image; for example, the binarization and final digitization of car’s tag 
numbers face these problems making them unfeasible or unreliable. FRIEB uses fuzzy reasoning 
based on heuristic techniques that compare the intensity value of each pixel in the image with the 



intensity levels of the immediate neighbors and decide if the intensity value of the pixel needs to be 
reduced or increased. Figure 3 shows the results of enhancing a text image via FRIEB prior to its 
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Figure 3. Effect of enhancing an image via FRIEB prior to binarization via FRAT and Huang- 
Wang techniques. 


binarization via adaptive binarization methods, such as FRAT and Huang-Wang methods. In figure 
3 the original image has a dark area on the right side of the image (due to uneven lighting 
distribution) that leads to poor binarization results using FRAT and Huang-Wang methods blocking 
the text located at the right side of the image. When the image is previously enhanced via FRIEB, 
the binarization of the image is dramatically enhanced as shown in figure 3. 


2.4 Artificial Neural Network (ANN) and Genetic Algorithm (GA) Systems 

ANN provides learning capabilities, and GA approach leads to robust learning results. The ANN 
scheme used along with FRED, FRAT, and FRIEB to add learning capabilities is commercially 
available (NeuroShell from Ward Systems Group, Inc.) and based on Probabilistic Neural Nets 
(PNN) formulated a little differently than conventional PNN; it is based on GA as opposed to a 
network structure. PNN’s instantly trained, unlike back propagation. In fact, they do not really 
train at all; they memorize, and the only freedom they have to generalize is to vary the smoothing 
factors. So the approach used in NeuroShell memorizes all of the patterns in the training set. Then 



it begins generating genetic individuals with GA. Each individual is a set of smoothing factors, 
which, when applied to PNN, makes a complete net. That net is tested against the test set, and the 
results become the GA fitness function. The GA generates the smoothing factors that perform the 
best against the test set, but all nets thus generated by the GA are trained on the training set 
(memorized). NeuroShell Classifier contains this new GA-PNN network paradigm. NeuroShell 
Classifier is spawned via DLL functions to perform training and firing of the GA-PNN 
classification net. 

3 PROTOTYPE APPLICATIONS 

3.1 Detection and Tracking of Moving Foreign Object Debris (FOD) within a Complex 
Background. 

A visual system for detecting and tracking multiple moving objects within a complex image 
background was developed and tested to detect potential FODs during the Space Shuttle liftoff. 
Images acquired and taped by a camera currently mounted at the Shuttle launch pad were used to 
test this real-time moving object detection and tracking system. 

The system was able to detect and completely track two potential FOD’s, as well as differentiate 
them from another moving object, such as ice flakes normally present during the Shuttle’s liftoff. 
The detection and tracking algorithm relies fundamentally on FRAT and a novel motion analysis on 
consecutive images. In contrast with existing moving object detection approaches, the FRAT-based 
object detection technique does not need image subtraction between consecutive image frames to 
detect moving objects. 

In the detection and tracking of moving FOD’s during the Space Shuttle liftoff, this prototype 
system uses FRAT to binarize the image and generate binary blobs that represent the detected 



objects in each image frame. Further analysis of the blobs on consecutives image frames allows the 
system to determine which blobs/objects are moving following a consistent path. The image 
background during the Space Shuttle liftoff is quite complex as moving objects that are not FOD’s 
(ice flakes, fume clouds, etc.) constantly appear at the scene following erratic moving paths. 

Figure 4a shows a FOD (indicated with a blue circle) that seems to be a screw falling from the top 
left (close to the Space Shuttle’s engine exhaust) toward the bottom left (close to the Space 
Shuttle’s stabilizer tail). The image background shown in this figure is quite complex as lots of 
additional moving objects that seem to be ice flakes produced during the disconnection of the 
Space Shuttle’s umbilical are constantly present as the FOD (screw) falls. 

Figure 4b shows a second object (indicated with a blue circle) falling from the top center (close to 
the Shuttle’s engine exhaust) toward the bottom left (close to the Shuttle’s stabilizer tail). Figure 4c 
shows the complete paths of both objects detected and tracked by the system in real time. 
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Figure 4a. Detection and tracking of first FOD. 






Figure 4c. Detection and tracking of both FODs. 


The FR-based system was able not only to detect actual FOD’s but also keep close track of them. 
The system was also able to process and execute motion detection features on every one of the 30 






frames per second generated by the image acquisition system. This SC-based visual system 
prototype received Honorable Mention during the 2003 NASA Software of the Year Competition. 

3.2 Visual Anomaly Detection on Slidewires 

A prototype application was developed to determine the feasibility of detecting anomalies via SC 
on slidewires used at the emergency egress system located at the NASA Space Shuttle launch pad. 
The visual anomaly detection approach was tested by detecting loose strands, a common and 
sometimes hidden and hard-to-detect anomaly class on slidewires. 

There are two Space Shuttle pads at KSC, each one having seven 1000- foot-long slidewires utilized 
to slide baskets that are used by the astronauts to evacuate the Space Shuttle launch pad in an 
emergency. Figure 5 shows astronauts drilling the emergency egress system at KSC; baskets are 
used to evacuate astronauts from the pad by sliding the baskets along the slidewires. 



Figure 5. Astronauts drilling the emergency egress 
system at the Space Shuttle launch pad. 

NASA built an inspection mechanism prototype called Cable and Line Inspection Mechanism 
(CLIM) equipped with four monochrome cameras to acquire images in four different angles along 
the slidewires. Figure 6 shows CLIM prototype being tested at the lab and the Space Shuttle launch 
pad. CLIM facilitates the visual inspection of the slidewires but still requires an operator to watch 



slidewire video images for many hours to spot and record anomalies. To eliminate human 
intervention that is prone to fail spotting and recording anomalies due to the tedious and repetitive 
nature of the job, an automated visual anomaly-detection system prototype was proposed and 
implemented on CLIM; this visual anomaly detection prototype system was developed via SC. 



Figure 6. CLIM being tested at the laboratory (right) and the launch pad (left). 


Using FRIEB, FRAT, and FRED the image goes through segmentation, object extraction, 
enhancement, filtering, and edginess detection; FRAT binarizes image preprocessed via FRIEB and 
FRED; a selective blob detector picks image regions with high edginess activity; morphological, 
geometrical, and fuzzy edginess values are computed for each region and used them as inputs to a 
Probability Neural Network (PNN) learning system. 



Figure 7. Loose strand detected by the system and indicated in separate windows. 



Preliminary tests were conducted to detect loose strands. Figure 7 displays the software front end 
developed at KSC to show the results of the firing part of the ANN-GA scheme. As illustrated in 
figure 4, the system also shows detected anomalies in a separate window. 

The window on the left comer of figure 4 displays the original image as acquired by the system in 
real time; the window on the bottom is generated by the system to display the image holding the 
anomaly indicated with a red rectangle. An additional window (white background) in figure 4 is 
also generated by the system to display the type of anomaly class found and its estimated 
confidence factor rated from 0 to 1 (being 1 the highest confidence). 

3.3. Object Recognition within an Image via Edge Detection and Morphological 
Properties Estimation. 

In contrast with conventional moving-object detection approaches, a general FR-based algorithm 
was developed to detect moving objects without the need of consecutive image frames. This 
general object detection algorithm was implemented to detect distant birds moving toward the 
Space Shuttle launch pad. The detection is performed on a single image frame by detecting edges 
via FRED to form blobs that are classified as birds based on key morphological properties 
(edginess and contrast) of the blobs. Only those blobs (formed by edge detection via FRED) with 
edginess and contrast values typical for birds are classified as birds. Figure 8 shows two image 
examples of bird detection via the proposed algorithm. In the first image example (top image) of 
figure 8, four distant birds are detected using a single image frame; the distant birds are 
considerably faded in the image as shown in cropped and magnified images of two of the four 
detected birds. In the second image example (bottom image) of figure 8, two birds are detected; the 
two detected birds are much closer and their detection happens to be more obvious. 




Figure 8. Examples of bird detection via the proposed SC-based scheme. 


In the top image of figure 8 four distant and hard-to-see birds are detected; two of them are cropped 
and magnified to the right. In the bottom image of the same figure, two closer birds and easier-to- 
see are detected; both are cropped and magnified to the right. 

Existing moving-object detection approaches are based on the difference between two consecutive 
image frames leading to the detection of objects located in different positions within two or more 
consecutive image frames. In this particular application (distant bird detection), the image 
difference approach does not work as expected since distant birds sometimes remain “suspended” 






and practically do not move during a considerable number of image frames. The motion of clouds, 
especially small clouds leads to false detection of birds if the difference between consecutives 
image frames is used. Image distortion caused by changing atmospheric conditions is a time-variant 
phenomenon leading to false detection of birds, as image pixel values of the image background do 
not remain constant. FRED-based edge detection scheme is used to form blobs; the blobs are 
classified based on key morphological properties that lead to the distinction and extraction of the 
blobs containing birds. 

Blob classification based only on edginess morphological value leads sometimes to false detection 
of birds; some blobs containing clouds yield similar edginess values expected for birds. Contrast, 
another morphological property of gray-scaled images, resolves this false detection problem. Figure 
9 illustrates how the addition of contrast value solves the false classification predicament that 
occurs sometimes when only edginess morphological value is used to classify gray-scaled image 
blobs formed via edge detection using FRED. 



Figure 9. Blob selection based only on edginess value leads sometimes to false detection of birds; 
some edge-based blobs containing clouds yield similar edginess values expected for 
bids. The contrast value resolves this false classification problem. 



Human vision is able to detect and distinguish particular objects (in our case birds) by seeing a 
single image regardless if the object is moving or not. The proposed algorithm also detects and 
distinguishes an object by analyzing a single image; in contrast with conventional, algorithms, it 
does not need consecutive image frames to detect moving objects. Edge detection is naturally 
performed by human vision to locate object boundaries, the different morphological properties of 
the areas under those detected edges lead to the recognition of the different objects within the scene 
or image. The proposed algorithm mimics the human vision (in a very primitive and basic way) in 
the use of morphological properties to distinguish objects on images. 

3.4 Space Shuttle Columbia Accident Investigation 

A set of FRAT-based imaging techniques was utilized to support the Space Shuttle Columbia 
accident investigation on the detection and trajectory-velocity estimation of the foam debris that hit 
the Orbiter’s wing. Two image acquisition sources, running at different frame rates and acquiring 
Shuttle’s images from different angle views and positions, were used to automatically detect the 
foam debris, determine its borders/edges, and estimate its 3D trajectory as well as its respective 3D 
speed. The detection of the foam debris and determination of its respective borders at each one of 
the 2D image frames acquired by two different image acquisition sources were performed by 
FRAT-based techniques. The image techniques used here were able to autonomously (not 
interactively) detect the blobs containing the foam debris from two different image acquisition 
sources, one a film camera with a relative good resolution and definition and the other a video 
camera with a poorer resolution and definition. 



Figure 10. Three Columbia images acquired at a consecutive sequence using two different image 
acquisition sources; a film camera acquired the first and third images and a video camera 
acquired the second image. FRAT allowed the detection and border determination of the foam 
in each image. Using the relative 2D location of the foam at each image and finding the 
interception of their 3D extension the 3D location of the foam can be estimated. 



Figure 11. Estimated 3D trajectory of the foam before hitting the Shuttle’s wing. 
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Figure 12. Estimated 3D foam velocity using the 3D position of figure 8 and 
the time recorded by the image acquisition sources. 


Figure 1 1 shows the estimated 3D trajectory of the foam using four sets of three consecutive 
images from two different image acquisition sources (film and video) as shown in figure 7. Figure 
9 shows the total 3D velocity computed using the trajectory shown in figure 8 and its respective 
time. These techniques also compute image blob geometrical properties such as area and center of 
mass. The center of mass of each debris blob was automatically calculated and used as the starting 
point of the respective 3D location line. A scaled 3D Shuttle model was used to manually 
determine the angle view of each 2D image acquired by the two image acquisition sources (film 
and video). 

3.5 Other applications 

The above FRED-based technologies have also proved their potential in enhancing x-ray images 
used to screen hard-covered items leading to a better visualization. Since the appearance of an x-ray 
film image is a plane in two dimensions of a structure built in three, the image is seen to contain a 
number of regions within the object superimposed on the background level. Therefore, to extract 
the contours of these different regions, it is necessary to have prior enhancement of the contrast 
levels among them. The FRED-based enhancement technique developed at NASA successfully 
deals with these problems. 

Preliminary tests to enhance x-ray images acquired at Johnson Space Center to screen pyrotechnic 
devices were conducted with very promising results as shown in figure 10. 

This FRED-based screening enhancement might be applied to x-ray images currently acquired to 
screen luggage at airports, containers in terminal ports, etc. Figure 1 1 shows another example of 
image enhancement of a saturated gray-scaled image saturated via FRED. Details hidden at 


saturated regions of both images in figures 10 and 1 1 are revealed as FRED extracts their edges, 
even the faintest ones using variable scanning windows sides. 



Figure 13. Enhancement and segmentation of x-ray images via FRED and FRAT respectively. 

A) original x-ray image of a pyrotechnic valve, B) Enhancement via FRED, C) 
Segmentation via FRAT. 



Figure 11. Enhancement of saturated gray-scaled image via FRED 
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